Repository navigation
Normalize database changes: verified baseline, SQL migration files, no more push - #215
Merged
Merged
Conversation
The old drizzle/ chain could not be applied: 81 SQL files vs 75 journal entries, duplicate numbers (0044, 0048, 0050, 0054, 0059, 0060), hand-written files outside the journal, a journal entry with no file and missing snapshots. Production was updated with drizzle-kit push, so drizzle.__drizzle_migrations there is empty, and generate on main wanted to re-create imap_mailbox_removals, which already exists. - Move the old SQL files, meta/, and the stale drizzle-kit pull output (drizzle/schema.ts, relations.ts) to drizzle/archive/ with a README; they are reference only and never applied. - drizzle.config.ts now writes to drizzle/migrations/. - Model the two hand-made production indexes from 0070 in lib/db/schema.ts (structured_emails_lower_recipient_idx, imap_mailbox_messages_mailbox_modseq_idx) so drizzle stops wanting to drop them. - 0000_baseline.sql: drizzle-kit generate of the full schema. - 0001_imap_notify_triggers.sql: custom migration with production's current IMAP notify functions and triggers (from pg_get_functiondef / pg_get_triggerdef), written idempotently. - scripts/migrate-domain.ts and scripts/lookup-user.ts import from @/lib/db/schema instead of the archived drizzle/schema. Verified: applied both migrations with drizzle's migrator to a fresh Postgres 17 and compared pg_dump --schema-only against a read-only dump of production. After ignoring column order, the dumps are identical except for the four *_merge_backup_20260929 tables, which are intentionally left out. drizzle-kit generate reports no changes and drizzle-kit check passes.
Removed: db-prepare (drizzle-kit generate + push against PROD_DATABASE_URL). Against a copy of production, push wanted to drop the four *_merge_backup_20260929 tables and the two hand-made indexes, and to drop and re-add three identical unique constraints (a drizzle-kit 0.31 bug). Removed: the drizzle-kit export-based local schema. It missed the IMAP triggers and hand-made indexes. scripts/local-dev/services.ts now applies drizzle/migrations to the local Docker database (dev:local and test:e2e). New scripts (scripts/db/, using drizzle-orm's node-postgres migrator and the existing pg dependency): - db:generate: drizzle-kit generate (pass --name / --custom). - db:check: drizzle-kit check, journal/file consistency, and a generate into a temp copy that fails if schema.ts changed without a migration. Needs no database and does not modify the repo. - db:migrate:local [db]: apply to the local Docker Postgres. - db:migrate:preview [branch]: pull the branch's Vercel Preview DATABASE_URL_UNPOOLED into a 0600 temp file (deleted after, never printed), refuse if it matches the production host (PROD_DATABASE_HOST / PROD_DATABASE_URL / Vercel production values; fails closed if none), apply. - db:migrate:prod: requires PROD_DATABASE_URL, prints host and pending migrations, requires typing the host or --yes. - db:baseline:prod / db:baseline:preview: one-time, records 0000 and 0001 in drizzle.__drizzle_migrations with the same hash (sha256 of the file) and created_at (journal when) drizzle's migrator writes. Dry run by default; refuses a non-empty table or a target missing any baseline column, index, enum value or IMAP trigger; --apply writes in a transaction. The runner refuses databases that have tables but no recorded migrations, and migrations older than the last applied one that are not recorded (edited or generated on an older base), which drizzle's migrator would skip silently. Verified on a throwaway Postgres 17: production's schema dump (no data) -> baseline dry run -> --apply -> rows identical to a migrator-built database -> db:migrate:prod reports nothing pending -> a dummy 0002 in a temp copy is the only pending migration and applies. Refusal paths checked (missing objects, non-empty table, edited migration, export-built database, preview host equal to production). bun run test:e2e: 23 pass before and after; the e2e database now has the 4 triggers and both indexes.
- CI: the unit job runs bun run db:check (no database needed). - AGENTS.md: replace "prompt the user to run drizzle-kit generate/push" and "don't run drizzle-kit commands" with a Database Changes section: schema.ts is the source of truth, every change ships as a numbered, reviewed SQL migration, never edit applied migrations, never push or export, apply to preview then production before deploying dependent code, additive first, production is applied by the user. Notes that pending migrations run in one transaction, so CREATE INDEX CONCURRENTLY can't be in a migration. - scripts/cleanup-duplicate-deliveries.ts no longer tells you to run drizzle-kit push. Verified: bun run db:check passes and fails when a column is added to schema.ts without a migration.
- PROD_DATABASE_HOST accepts a bare host or a URL, strips the port and the Neon -pooler suffix, and rejects anything still containing / : or @. All production/preview host comparisons use the same normalisation. - Preview resolution also pulls the environment-wide Preview values; if the branch's database host equals the shared one (vercel env pull falls back when a branch has no scoped vars), refuse unless --allow-shared-preview. The checks live in checkPreviewDatabase so they can be exercised without Vercel. - Baseline verification also checks PK, unique and FK constraint names from the snapshot against pg_constraint. A target without the IMAP triggers now records only 0000 and leaves the idempotent 0001 pending for db:migrate, so export-built local databases can be baselined. - db:baseline:local added; the "tables but no history" error points to the baseline commands. - db:migrate fails on edited/unrecorded older migrations right after reading status, before any prompt, even when nothing is pending. - Prod confirmation takes --yes=<host>; bare --yes is rejected. - applyMigrations holds a session-level pg_advisory_lock around drizzle's migrator (which opens its own transaction); baseline takes pg_advisory_xact_lock in its transaction. - drizzle.config.ts has no dbCredentials, so drizzle-kit push/pull/studio fail closed; generate and check do not need a database. - AGENTS.md: enum ADD VALUE trap, db:baseline:preview for preview branches forked before the baseline, "no raw SQL" scoped to application code. Verified on throwaway Postgres 17: prod schema dump (read-only) baseline dry run and --apply with --yes=<host> (bare and wrong --yes refused), then nothing pending; a copy missing a unique constraint and an FK is refused with both listed; an edited applied migration fails before any prompt; 3 concurrent applyMigrations apply once; export-built DB baselines 0000 and db:migrate applies 0001. Host guards checked with mocked env values. Scoped tsc over the changed files: 0 errors. db:check passes, test:unit 185 pass, test:e2e 23 pass.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
R44VC0RP
added a commit
that referenced
this pull request
Oct 21, 2026
Main restarted the Drizzle history from a verified production baseline (drizzle/migrations). This branch's old-chain drizzle/0075_jev_classification.sql and meta/0075_snapshot.json are dropped and the same change is regenerated with bun run db:generate as drizzle/migrations/0002_jev_classification.sql (ALTER TABLE structured_emails ADD COLUMN jev_classification text). Verified: db:check passes; guard and retry tests 77/77.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces
drizzle-kit pushto production with a normal workflow of plain SQL change files, starting from a baseline verified against production.Why
A read-only audit on 2026-10-21 found:
0044,0048,0050,0054,0059,0060), and orphan hand-written files weren't in the journal.0047_thin_wind_dancer), and snapshots were missing.drizzle.__drizzle_migrationswas empty.bun run db-prepare(generate +drizzle-kit push) was dangerous. Run against a copy of production, it wanted to:*_merge_backup_20260929tables (about 730 MB);structured_emails.drizzle-kit generateon main was stale. It emittedimap_mailbox_removals, which already exists in production.drizzle-kit export, so they had no IMAP notify triggers and no hand-made indexes.Production itself matches
lib/db/schema.tsexactly, apart from those indexes, triggers and backup tables.What changed
History restarted (
d05b5322):drizzle/archive/, with a README. They're for reference only and are never applied.schema.ts.drizzle/migrations/0000_baseline.sql, plus0001_imap_notify_triggers.sqlwith production's current function and trigger definitions (safe to re-run).Scripts (
6b022153,30c5291f), inscripts/db/:db:generatedb:checkschema.tschanged without a migration; runs in CI with no databasedb:migrate:localdb:migrate:preview <branch>db:migrate:prodPROD_DATABASE_URLand--yes=<host>or typing the hostdb:baseline:{prod,preview,local}Other changes in these commits:
db:migratefails before any prompt if an already-applied migration was edited.drizzle.config.tsno longer has database credentials, so an accidentaldrizzle-kit push/studiofails instead of connecting.db-prepareis removed.Rules (
5bd68e45): a new "Database Changes" section in AGENTS.md:push;CREATE INDEX CONCURRENTLY, and a new enum value can't be used in the same run);CI runs
bun run db:check.Verification
pg_dump --schema-onlywith a read-only dump of production. The only difference is the 4 backup tables, left out on purpose. After that,drizzle-kit generatereports no changes.created_atvalues that drizzle-orm 0.43.1's migrator writes. After it, migrate reports nothing pending, and a dummy0002is the only pending migration.--yes, and the shared preview fallback.bun run test:e2e23/23 andbun run test:unit185/185, before and after.db:checkpasses, and fails whenschema.tschanges without a migration.After merging (run by the user, once)
Existing local databases: run
bun run db:baseline:local, or reset withbun run dev:local reset.Effect on #214
#214 adds
drizzle/0075_jev_classification.sqlon the old history. Once this merges:bun run db:generate --name jev_classification; it should produce0002.Not in this PR
The four
*_merge_backup_20260929tables stay in production. Drop them separately once the Sept 29 merge is confirmed final.Note
High Risk
This changes how production and preview databases are updated; a mistaken migrate/baseline on the wrong host or skipping the one-time prod baseline could break IMAP or leave migration history inconsistent.
Overview
Replaces
drizzle-kit pushand the broken pre-October migration tree with a replayable workflow:lib/db/schema.tsstays the source of truth, and every change ships as numbered SQL underdrizzle/migrations/.Migration history is restarted: prior files move to
drizzle/archive/(reference only). Live chain is0000_baseline.sql(production schema as of 2026-10-21) and0001_imap_notify_triggers.sql(idempotent IMAPpg_notifytriggers).drizzle.config.tswrites todrizzle/migrationsand drops DB credentials so accidental push/studio cannot connect.Tooling: removes
db-prepare; addsdb:generate,db:check(CI unit job),db:migrate:{local,preview,prod}, anddb:baseline:{local,preview,prod}viascripts/db/(journal validation, advisory locks, prod host confirmation, preview vs production guards). Local dev and e2e apply pending migrations instead ofdrizzle-kit export.Schema: documents two production-only indexes in
schema.ts(structured_emails_lower_recipient_idx,imap_mailbox_messages_mailbox_modseq_idx).AGENTS.mddocuments the new rules; a few scripts import@/lib/db/schemainstead of the olddrizzle/schemapath.Reviewed by Cursor Bugbot for commit 30c5291. Bugbot is set up for automated code reviews on this repo. Configure here.